<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" >

<title>Pytorch计算图内存释放问题 | KALUO</title>
<meta name="description" content="温故而知新">

<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">

<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.2/css/all.css" integrity="sha384-fnmOCqbTlWIlj8LyTjo7mOUStjsKC4pOpQbqyi7RrhN7udi9RwhKkMHpvLbHG9Sr" crossorigin="anonymous">
<link rel="shortcut icon" href="https://kaluo-zz.github.io/favicon.ico?v=1591084625855">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.10.0/katex.min.css">
<link rel="stylesheet" href="https://kaluo-zz.github.io/styles/main.css">



<script src="https://cdn.jsdelivr.net/npm/vue/dist/vue.js"></script>
<script src="https://cdn.bootcss.com/highlight.js/9.12.0/highlight.min.js"></script>

<link rel="stylesheet" href="https://unpkg.com/aos@next/dist/aos.css" />



  </head>
  <body>
    <div id="app" class="main">

      <div class="sidebar" :class="{ 'full-height': menuVisible }">
  <div class="top-container" data-aos="fade-right">
    <div class="top-header-container">
      <a class="site-title-container" href="https://kaluo-zz.github.io">
        <img src="https://kaluo-zz.github.io/images/avatar.png?v=1591084625855" class="site-logo">
        <h1 class="site-title">KALUO</h1>
      </a>
      <div class="menu-btn" @click="menuVisible = !menuVisible">
        <div class="line"></div>
      </div>
    </div>
    <div>
      
        
          <a href="/" class="site-nav">
            首页
          </a>
        
      
        
          <a href="/archives" class="site-nav">
            归档
          </a>
        
      
        
          <a href="/tags" class="site-nav">
            标签
          </a>
        
      
        
          <a href="/post/about" class="site-nav">
            关于
          </a>
        
      
    </div>
  </div>
  <div class="bottom-container" data-aos="flip-up" data-aos-offset="0">
    <div class="social-container">
      
        
      
        
      
        
      
        
      
        
      
    </div>
    <div class="site-description">
      温故而知新
    </div>
    <div class="site-footer">
      Powered by <a href="https://github.com/getgridea/gridea" target="_blank">Gridea</a> | <a class="rss" href="https://kaluo-zz.github.io/atom.xml" target="_blank">RSS</a>
    </div>
  </div>
</div>


      <div class="main-container">
        <div class="content-container" data-aos="fade-up">
          <div class="post-detail">
            <h2 class="post-title">Pytorch计算图内存释放问题</h2>
            <div class="post-date">2019-07-01</div>
            
            <div class="post-content">
              <p>pytoch构建的计算图是动态图，为了节约内存，所以每次一轮迭代完之后计算图就被在内存释放，所以当你想要多次<strong>backward</strong>时候就会报如下错：</p>
<pre><code class="language-python">&gt;&gt;&gt; from torch import nn
&gt;&gt;&gt; net = nn.Linear(3, 4)
&gt;&gt;&gt; input = Variable(torch.randn(2, 3), requires_grad=True)
&gt;&gt;&gt; output = net(input)
&gt;&gt;&gt; loss = torch.sum(output)
&gt;&gt;&gt; loss.backward()  # 到这计算图已经结束，计算图被释放了
&gt;&gt;&gt; loss.backward()
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;/home/kaluo/.local/lib/python2.7/site-packages/torch/tensor.py&quot;, line 93, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File &quot;/home/kaluo/.local/lib/python2.7/site-packages/torch/autograd/__init__.py&quot;, line 90, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>之所以会报这个错，因为计算图在内存中已经被释放。但是，如果你需要多次<strong>backward</strong>只需要在第一次反向传播时候添加一个标识，如下：</p>
<pre><code class="language-python">&gt;&gt;&gt; input = Variable(torch.randn(2, 3), requires_grad=True)
&gt;&gt;&gt; output = net(input)
&gt;&gt;&gt; loss = torch.sum(output)
&gt;&gt;&gt; loss.backward(retain_graph=True) # 添加retain_graph=True标识，让计算图不被立即释放
&gt;&gt;&gt; loss.backward()

</code></pre>
<p>经过多次backward后的叶节点梯度会直接累加，所以我们一般会将其清零。</p>
<pre><code class="language-python">&gt;&gt;&gt; x = Variable(torch.FloatTensor([[1, 2]]), requires_grad=True)
&gt;&gt;&gt; y = Variable(torch.FloatTensor([[3, 4],
...                                 [5, 6]]))
&gt;&gt;&gt; loss = torch.mm(x, y)
&gt;&gt;&gt; loss.backward(torch.FloatTensor([[1, 0]]), retain_graph=True)
&gt;&gt;&gt; print(x.grad.data)
tensor([[3., 5.]])
&gt;&gt;&gt; x.grad.data.zero_() # 最后的梯度会累加到叶节点，所以叶节点清零
tensor([[0., 0.]])
&gt;&gt;&gt; loss.backward(torch.FloatTensor([[0, 1]]))
&gt;&gt;&gt; print(x.grad.data)
tensor([[4., 6.]])

</code></pre>

            </div>
            
              <div class="tag-container">
                
                  <a href="https://kaluo-zz.github.io/tag/rT7AxA2VY" class="tag">
                    pytorch
                  </a>
                
                  <a href="https://kaluo-zz.github.io/tag/-AEi1Fqbq" class="tag">
                    计算图
                  </a>
                
              </div>
            
            
              <div class="next-post">
                <div class="next">下一篇</div>
                <a href="https://kaluo-zz.github.io/post/linux-gua-zai-xin-ying-pan">
                  <h3 class="post-title">
                    Linux挂载新硬盘
                  </h3>
                </a>
              </div>
            

            

          </div>

        </div>
      </div>
    </div>

    <script src="https://unpkg.com/aos@next/dist/aos.js"></script>

<script type="application/javascript">

AOS.init();

hljs.initHighlightingOnLoad()

var app = new Vue({
  el: '#app',
  data: {
    menuVisible: false,
  },
})

</script>




  </body>
</html>
